Papers with Masked language models
Deriving Language Models from Masked Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Masked language models do not define an explicit distribution over language, but they assume that tokens masked out are conditionally independent given the unmasked tokens. |
| Approach: | They propose to use a set of MLM's unary conditionals to construct a fully-connected Markov random field over the input to deduce an explicit joint distribution from MLMs. |
| Outcome: | The proposed method outperforms existing Markov random field-based approaches and outperformed the original model's conditionals. |
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained MLMs produce an anisotropic distribution of token representations . this is not ideal for tasks that require discriminative semantic meanings of distinct tokens - a problem that exists in pre-training models . |
| Approach: | They propose a continual pre-training approach that encourages BERT to learn an isotropic distribution of token representations. |
| Outcome: | The proposed approach improves on a wide range of English and Chinese benchmarks. |
Should You Mask 15% in Masked Language Modeling? (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language models (MLMs) traditionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations. |
| Approach: | They revisit the 15% masking rate of MLMs to examine the role of masking in linguistic training. |
| Outcome: | The proposed masking rate outperforms BERT-large size models on GLUE and SQUAD while maintaining 95% accuracy. |
Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks (2021.naacl-main)
Copied to clipboard
| Challenge: | Masked language models have become the de facto standard when processing text . however, these models are evaluated in a monolingual setting only . |
| Approach: | They propose a language-independent entity prediction task as an intermediate training procedure to ground word representations on entity semantics and bridge the gap between different languages. |
| Outcome: | The proposed approach bridges the gap between word representations and knowledge graphs by using a shared vocabulary of entities. |
Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language models have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). |
| Approach: | They propose a pretraining task that trains MLMs on anaphoric relations with explicit supervision and a finetuning method that remedies a notorious discrepancy. |
| Outcome: | The proposed method improves zero anaphora resolution in Japanese ZAR . it uses a pretrain task and finetuning task to correct the discrepancy . |
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has obtained strong zero-shot results by prompting language models. |
| Approach: | They propose a mining-based approach that uses regular expressions to mine labeled examples from unlabeled corpora and fine tune a pretrained model. |
| Outcome: | The proposed method outperforms prompting on a wide range of tasks when using comparable templates. |
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior attempts at measuring leakage of MLMs via membership inference attacks have been inconclusive, implying potential robustness of Mlms to privacy attacks. |
| Approach: | They propose a stronger membership inference attack based on likelihood ratio hypothesis testing that involves an additional MLM to more accurately quantify the privacy risks of memorization in MLMs. |
| Outcome: | The proposed attack improves the AUC of prior membership inference attacks from 0.66 to an alarmingly high 0.90 level on models trained on medical notes. |